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SECTION  I 
INTRODUCTION 


Problems  of  estimation,  control,  and  qualitative  analysis  for  noisy 
engineering  systems  are  central  to  the  Control  System  Test  Vehicle  (CSTV) 
project.  The  work  was  concerned  with  an  approach  to  these  problems  via 
the  transition  probability  perspective  which  has  promise  for  some  problems 
which  fall  outside  the  domain  of  the  linear-quadratic-Gaussian  techniques. 

In  particular,  the  methods  are  applicable  to  the  nonlinear  models  of  the 
CSTV. 

The  relevant  part  of  the  literature  on  Markov  processes  is  unusually 
profound,  requiring  knowledge  of  probability  theory  and  dynamical  systems. 

This  report  attempts  to  explain  that  part  of  the  theory  which  is  applicable 
to  the  CSTV  models  by  considering  a  series  of  examples  and  special  cases. 

For  instance,  the  general  theory  treats  the  model 

dX(t)  =  b[X(t)J  dt  +  o[X(t)]  dW(t) 

with  the  Ito  integral  as  opposed  to  the  Riemann  integral  of  ordinary  calculus 
because  a(x)  depends  on  the  state.1  The  coefficient  o(x)  is  constant  for  the 
CSTV  models  affording  a  great  simplification.  We  will  only  treat  this  special 
case . 


The  transition  probability  approach  is  relevant  in  that  it  allows  for 
refinements  in  current  techniques;  i.e.,  in  estimating  the  steady-state  dis¬ 
tribution  of  the  state  vector,  in  nonlinear  parameter  estimation,  and  in 
control.  These  refinements  are  important  because  the  nonlinear  models 
might  exhibit  complex  behavior  such  as  multiple  modes  with  a  resultant 
quasi-periodic  motion  which  cannot  be  dealt  with  using  the  LQG  techniques. 

The  material  in  this  report  is  presented  informally  in  terms  of 
examples  with  the  intention  of  illustrating  the  nature  of  the  problems 
and  techniques  of  solution;  we  are  trying  to  present  an  outline  or.  road 
map  rather  than  a  detailed  survey. 


*Gelb,  A.  (Ed),  Applied  Optimal  Estimation,  The  M.  I.  T.  Press,  1974. 
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ORGANIZATION 

Section  II  begins  with  a  discussion  of  the  general  mathematical  back¬ 
ground.  Since  we  will  consider  a  part  of  the  general  theory  of  Markov 
processes,  we  will  attempt  to  define  the  distinctions  between  the  two 
approaches  to  Markov  processes;  i.e.,  the  sample  path  approach  to  which 
engineering  models  most  naturally  belong  and  the  transition  probability 
approach. 

The  partial  differential  equations  of  Kolmogorov  which  are  of  particu¬ 
lar  mathematical  importance  in  this  report  are  discussed  in  Section  III.  A 
brief  derivation  of  the  equations  and  a  discussion  of  the  relationship  between 
the  partial  differential  equations  and  stochastic  differential  equations  (the 
engineering  models)  are  included.  The  problems  of  existence  of  solutions 
(well  posedness)  and  asymptotic  results  are  mentioned. 

The  problems  of  parameter  estimation,  the  most  developed  application 
of  the  methods  outlined  in  Section  III,  are  discussed  in  Section  IV.  Sec¬ 
tion  V  discusses  the  qualitative  analysis  of  noisy  system  and  Section  VI 
the  control  of  noisy  systems. 


SECTION  II 
BACKGROUND 


TRANSITION  PROBABILITY  FUNCTIONS 

In  systems  theory,  the  study  of  a  system  most  often  begins  with  a 
model,  typically  X  =  AX  +  BU.  In  the  theory  of  stochastic  processes,  some 
of  which  have  models  of  the  form  dX(t)  =  b[X(t)]  dt  +  O  dW(t),  the  study 
is  much  more  likely  to  begin  with  functions  defined  on  the  sample  paths 
{X(t),t>0}.  For  instance,  the  system  concept  of  state  (the  Markov  property: 
given  the  present,  the  future  is  independent  of  the  past2)  is  defined  in  terms 
of  conditional  distributions  on  {X(t),t>0}  rather  than  as  a  property  of  some 
model.  Markov  processes  are  usually  described  in  the  literature3  4  5  by 
transition  probability  functions. 


^Parzen,  E.,  Stochastic  Processes,  Holden-Day,  1962. 

3Dynkin,  Y.  B.  ,  Markov  Process,  Vols.  I  and  II,  Springer-Verlag,  196r>. 

4Fuller,  W. ,  An  Introduction  to  Probability  Theory  and  Its  Applications, 
Vol.  I,  Third  Edition,  John  Wiley  &  Sons,  1968. 


sGikhman,  I.  I.  and  Skorohod,  A.  V.,  The  Theory  of  Stochastic  Processes, 
Vol.  I  through  III,  Springer-Verlag,  1974,  1975,  1979. 
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A  function  P(E,t  |x,tg)  is  a  transition  probability  function  on  a  Markov 
process  {X(t),t>0}  provided  P(E,t|x,tg)  is  the  conditional  probability  that 
X(t)  belongs  to  E,  given  that  at  time  tQ  <  t  we  have  X(tQ)  =  x.  The  process 
is  stationary  or  homogeneous  in  time  if  P(E,t|x,  tQ)  =  P(E ,t-tQ | x ,0) .  We 
are  only  concerned  with  this  case  and  write  P(E,t|x)  in  the  place  of  P(E,t|x,0). 


The  transition  probability  function  satisfies  the  Chapman-Kolmogorov 
equation6 


P(E , t  +  s|x)  =  f  P(E,t|y)  PCdy,  s|x). 

R 

If  a  transition  density  p(y,t|x)  exists,  i.e.,  if  P(E,t|x)  =  f  p(y,t|x)dy, 
then  E 

p(y,t  +  s | x)  =  f  p(y,t|z)  p(z,s|x)  dz. 

R 

Note  that  if  p(*,t|x)  is  known  for  0  <  t  <  £,  then  p(*,t|x)  can  be  found  for 
all  t  by  iterating.  Under  suitable  continuity  conditions  and  with  proper 
interpretations  p(*,o|x)  and  3/8t(p(* ,t|x)) j t=0  completely  determine  p(*,t|x) 

for  all  t.7  Furthermore,  p(*,o|x)  is  always  the  same;  viz,  p(y,0|x)  =  6(y-x) . 

There  is  extensive  literature  on  Markov  processes  dealing  with  both  the 
mathematical  foundations  and  applications.  The  literature  can  be  considered 
to  be  composed  of  three  parts.2 

1.  The  study  of  time  dependent  behavior:  to  find  the  transition 
probability  function  by  finding  and  solving  (differential, 
integral,  or  other  functional)  equations  which  is  satisfies. 

2.  The  study  of  long  run  behavior:  to  find  conditions  under  which 
a  steady  state  exists;  i.e.,  lim  P(E,t[x)  =  n(E). 

t-*» 

3.  The  study  of  the  qualitative  behavior:  to  examine  the  proba¬ 
bility  distribution  of  the  amount  of  time  the  system  spends  in 
various  states 


2ibid. 

8Arnold,  L. ,  Stochastic  Differential  Equations:  Theory  and  Applications, 
Interscience,  1974. 

7Wong,  L. ,  Stochastic  Processes  in  Information  and  Dynamic  Systems,  McGraw- 
Hill,  1971. 
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and  the  length  of  time  required  for  the  system  to  pass  from  one  set  of  states 
to  another. 

The  first  two  areas  will  be  discussed  in  Section  III  and  the  third  area  in 
Section  V. 

The  transition  probability  function  approach  to  engineering  problems  can 
be  justified  in  three  ways. 

1.  This  approach  provides  a  distinctive  modeling  technique. 

2.  The  models  are  deterministic  dynamical  systems  which  allow 
for  application  of  classical  methods. 

3.  This  approach  leads  to  more  sophisticated  models  than  the 
sample  path  approach. 


SAMPLE  PATH  VERSUS  TRANSITION  PROBABILITIES 

One  approach  to  modeling  engineering  systems  operating  in  noisy  environ¬ 
ments  or  systems  for  which  only  noisy  measurements  can  be  made  is  to  modify 
deterministic  models,  i.e.,  differential  equation  models  assuming  complete 
knowledge,  by  replacing  parameters  with  stochastic  processes  to  obtain  sto¬ 
chastic  differential  equations.8  There  is  a  lot  of  flexibility  in  this  approch 
even  if  it  is  restricted  to  the  Wiener  process  (the  "derivative"  of  a  Wiener 
process  is  white  noise)  or  to  processes  obtained  from  a  Wiener  process.  There 
is  some  controversy  in  the  literature  on  the  appropriateness  of  the  resulting 
models,8  9  but  the  issues  of  the  controversy  do  not  seem  to  apply  to  the  CSTV 
models.  For  our  models,  the  noise  terms  are  state  independent. 

One  approach  for  obtaining  transition  probability  models  begins  with  a 
discrete  model  and  produces  the  final  model  through  a  limit.  The  canonical 
example  of  this  approach  is  the  diffusion  equation  model  of  Brownian  motion 
obtained  from  a  random  walk  model.10  Another  example  is  that  of  a  population 
model  obtained  from  a  discrete  birth/death  process  model.11  There  is  a  feel¬ 
ing  in  this  approach  that  the  discrete  model  is  "microscopic"  and  the  con¬ 
tinuous  model  obtained  as  a  limit  is  "macroscopic."  We  will  see  in  Section  III 


8McShane,  E.  J. ,  Stochastic  Calculus  and  Stochastic  Models,  Academic  Press, 
1974. 

9Gray,  A.  H.  and  Caughey,  J.  K. ,  A  Controversy  in  Problems  Involving  Random 
Parametric  Excitation,  J.  Math  and  Phys . ,  44  (1975),  pp.  288-296. 

10Wax,  N.,  Selected  Papers  on  Noise  and  Stochastic  Processes,  Dover  Publica¬ 
tions,  1954. 

11Bartlett,  M.  S.,  Stochastic  Population  Models  in  Ecology  and  Epidemiology, 
Methune,  London,  I960. 
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how  to  relate  the  stochastic  differential  equation  models  with  the  transition 
probability  models. 

The  stochastic  differential  equation  models  are  often  nonlinear.  Although 
we  have  satisfactory  computational  methods  for  models  which  are  linear,  methods 
for  the  nonlinear  models  are  much  less  developed.1  Often  our  only  option  in 
analyzing  these  nonlinear  models  is  simulation.  While  simulation  is  a  powerful 
technique,  it  is  cumbersome.  On  the  other  hand,  the  transition  probability 
models  are  deterministic  dynamical  systems.  Frequently,  the  resulting  model 
is  a  parabolic  or  elliptic  partial  differential  equation  of  a  type  that  has 
been  studied  for  decades  and  for  which  numerical  methods  are  highly  developed.12 

The  transition  probability  approach  requires  a  lot  of  mathematical  sophis¬ 
tication.  Many  deep  results  from  analysis,  probability  theory,  and  dynamical 
systems  are  called  on.  However,  the  resulting  models  and  analysis  are  both 
very  elegant  and  powerful. 

Finally,  the  literature  seems  to  be  divided  into  two  camps:  those  iden¬ 
tified  with  the  sample  path  approach  and  those  identified  with  the  transition 
probability  approach.  Since  both  approaches  are  attacks  on  the  same  problem, 
the  analysis  of  noisy  systems,  we  should  be  able  to  make  use  of  both. 


SECTION  III 

THE  PARTIAL  DIFFERENTIAL  EQUATIONS  OF  KOLMOGOROV 


RELATING  THE  TWO  APPROACHES 


Suppose  that  dX(t)  =  b[X(t)]  dt  +  a  dW(t);  i.e., 

t+At 

X(t  +  At)  -  X(t)  =  /  b[X(s) ]  ds  +  a  [W(t  +  At)  -  W(t)] 
t 

=  b[X(t) ]  At  +  a  |W(t  +  At)  -  W(t)]. 


Then 


EtXCt  + .  At)_-_X(t))  _  E[b(x(t) ] 


00 

=  /  b(y)  p(y,t|x)dy, 
-00 


12Bers,  L. ,  John,  F. ,  and  Schechter,  M.,  Lectures  in  Applied  Mathematics, 
Vol.  Ill,  Partial  Differential  Equations,  Interscience,  1964. 
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where  te(t,  t  +  At)  and  X(t)  =  x.  Hence, 


E  [X(t  +  At)  -  X(t)] 
At 


■*  b(x)  as  At  ^  0. 


Also, 

E([X(t  +  At)  -  X(t)]  (X(t  +  At)  -  X(t))T) 

=  E(b[X(t) ]  b[X(t)]T)  (At)2  +  E(b(X(t))(W(t  +  At)  -  W(t)))  aAt 
+  a2  E([W(t  +  At)  -  W(t)]  [W(t  +  At)  -  W(t)]T). 

Hence 

E({X(t  •**  At)  -  X(t) }  [X(t  +  At)  -  X(t)]T)  ^  o2  ag  Afc  ^  Q 


Note  that 

E(X(t  +  At)  -  X(t)) _ 1 

At  ~  At 


and 


Qo 

/  (y  -  *)  p(y,At|x)dy 

-QO 


E([X(t  +  At)  -  X(t)][X(t  +  At)  -  X(t)]T) 

At 


=  At  ^  (y  -  *Hy  -  x)Tp(y,At|x)  dy. 

—00 

This  computation  leads  to  the  following  definition. 13  A  Markov  process 
(X(t),  t  >  0}  with  continuous  sample  paths  is  called  a  diffusion  process14 
if  its  transition  density  p(y,t|x)  satisfies  for  E>0  as  t-*0 

1.  f  S  P(y.t)x)  dy-*0 

*  |y-x|>e 

2.  ;  J  (y  -  x)p(y,t|x)  dy  -*■  b(x)  (drift) 

|y-*|<e 


13Kolmogorov,  A. ,  Uber  die  Analytische  Methoden  in  der  Wahrscheinlich- 
keitsrechnung,  Math.  Ann.  104,  pp.  415-458,  1931. 

14Stroock,  0.  and  Varadhau,  S.  R.  S.,  Diffusion  Processes  with  Boundary  Con¬ 
ditions,  Coon.  Pure  Appl.  Math.,  24(1971),  pp.  147-225. 


6 
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and 


1  T  l 

3.  -  f  (y  -  x)(y  -  x)  p(y,t[x)  dy  -*■  a(x)  (dispersion 

|y-x|<e 

We  are  concerned  with  diffusions  for  which  the  dispersion  coefficient 
is  not  state  dependent;  i.e.,  a(x)  is  constant.  However,  for  the  moment 
let  us  consider  the  general  one-dimensional  case. 

For  sufficiently  nice  functions  f(x),  we  have 


oo 

u(t,x)  =  f  f (y)  p(y,t| x)  dy 
-oo 


which  is  defined  for  all  t>0  and  -»  <  x  <  <*>.  We  want  to  show  that  u  solves 
the  backward  equation13 

3u  ,  ,  v  32u  .  .  ,  .  3u 

U  '  a(x)  S*  *  b(x)  ai 


u(0,x)  =  f(x). 

Recalling15  the  Chapman- Kolmogorov  equation 

oo 

p(y,t  +  Atjx)  =  /  p(y,t|z)  p(z,At|x)  dz  , 


we  have 


u(t  +  At ,x)  -  u(t,x)  =  X  f (y)  {  s  p(y,t|  z)p(z,At  x)  dz  -  p(y,t|  x)}  dy 

-00  -00 

=  S  f(y)  {  S  {p(y»t|x)  +  (2  .  x) 

—00  —  00 

+  h  9  ^z  "  x^2  +  °^z  ~  xf2^  p(z,At|  x)  dz  -  p(y,t|  x) }  dy 


13ibid. 

lsBrannan,  J. ,  Classroom  Notes,  Department  of  Mathematical  Sciences, 
Clemson  University,  1980-81. 
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oo  oo  i 

=  S  f Cy) {/  (z  -  x)p(z,At|x)  dz 


00 

+  s  (z 
-00 


x)2  p  (z,At  |x)  dy 


j.  9zpCy>  <-  I*) 

*  ax* 


00 

+  S  0(  |z  -  x  |2)  p(z,At  lx)  dz}  dy. 
-oo 


Hence 


u(t  +  At,x)  -  u(t,x) 
At 


”  f(y){b(x)  *  >,  a(x)  dy 


as  At  -»  0. 

On  the  other  hand, 

Mijil  *  °}  f(y)  at.(y.t|x)  dy 
—00 

and 

.  *  f(y)  SlLtojElxl  dx 
-00 

Therefore 

( t , x )  =  b(x)  (t,x)  +  *sa(x)  (t,x). 

Since  p(y,0|x)  =  6(y  -  x)  we  also  have  u(0,x)  =  f(x).  The  name  "backward 
equation"  evidently  comes  from  the  fact  that  y  and  t  are  held  constant  and 
we  are  looking  backward  in  time  to  differentiate  p(y,t|x)  with  respect  to 
x.15 


The  general  backward  equation  of  Kolmogorov  is 
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The  probabilistic  interpretation  of  u  is  that  u(t,x)  is  the  expected  value 
of  f(X(t))  given  that  X(0)  =  x. 

We  proceed  now  with  the  forward  equation.15  For  the  backward  equation 
the  n-dimensional  case  is  essentially  the  same  as  the  one-dimensional  case. 
For  the  forward  equation  the  n-dimensional  case  is  essentially  the  same  as 
the  two-dimensional  case.  So  we  consider  n  =  2. 

For  a  sufficiently  nice  function  uQ(z)  =  u^(z  ,z2)  from  R2  to  R1  the 
Chapman-Kolmogorov  equation  yields 


« 

J 

-00 

00 

=  J 


-00  -  00  -00  -oo 


00 

=  s 


where 


u(t,y)  =  / 
-00 


Further, 


_3 

at 


00 

J 


-00  -00 


f  uQ(z)  p(z,t  +  t jx)  dzJdz2 
-00 

00  00  00 

Iff  uQ(z)  Pte.tly)  pCy,x]  x)  dyJdy2dz1dz2 

J  u(t,y)  p(y,t|x)  dyJdy2, 

-00 

00 

/  uQ(z)  p(z.t|y)  dzJdz2. 

-00 

00 

f  uQ(z)  P(z,t  +  x |x)  dz1dz2 


00  00 


=  J  J  uQ(z)  p(z,t  +  x|x)  dZjdz2 


-00  -00 


OD 

; 


-00  -00 


0 

J  uq(z)  p(z,t  +  t|x)  dZjdz^ 


00 

J 

-00 

00 

; 


3 

S  qI  u(t,y)  p(y,l|x)  dy^yj 


f  {hi  a,,(y)  ^7,(L',y)  ♦  I  b, (y)  p(y,i|x)  dy 

j=l  yj 


*  •  ij  9y.ay . 

1  ,J=1  1  J 
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We  next  apply  integration  by  parts  which  lor  each  i  and  j 
/  I  a  (y)  P(y,l|x)  dy  dy2 

-Oo  -Oo  J  J  i  7  j 


y  =  oo 

y  ■  =  -,0 


s  fa  (y)  Hv,y}  p(y>Tlx> 

yj 

' s  8u^3y  ~  4'.  (aij(y)p^y>llx)(Jyi5  dyj 


=  -  J  {u(y , t)  X-  (a  (y)p(y , I | x) ) 

-oo  y  i  J 


-  I  u(y,t)  3V  v“  (aii(y)p(y,x|x)dy  }  dy 

-oo  ^  i  *  j  **  J 


y  n  oo 
=.  -oo 

j 


a2 

f  f  u(y,t)  g—Q—  (a.  (y)p(y,T|x)>  dy]dy2 

-oo  -oo  ■*  i  j  ^  ^ 


A  similar  process  yields  for  each  j 


So  we  have 


00  00 


S  f  b-(y)  gTT  u(y>b)p(y»x|x)  dytdy2 

—00  —00  J  *  i 


oo  oo 


■  J  X  (b  .  (y)p(y , T | x) )  dy  dy 

-oo  -oo  >j  J 


00  00 


J  J  u0(y)  §7  p(y»t  +  t|x)  dy  fly 
-00  -00 

oo  oo  2  2 

J  J  u(t,y)  {*&  *  *— -g--  (a. . (y)p(y,T|x)) 

— oo  —oo  i  j  j  —  1  ^  i  *  j 


'  1  3yT  (bj(y)P(y»T'x))J  dyjdy2  . 


As  t  +  0,  u(t,y)  -*  Ugly).  Hence 


yields 


10 
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f  S  u0(y^a7  p(y>Tlx)  "  h  z  (aii(y)p(y»t|x)) 
-os  -oo  i ,  j  =  l  J 


+  2  g~  (b.(y)p(y,T  |x))}  dy:dy  =  0 
j=l  Yj  3 


for  all  sufficiently  nice  u^(y);  i.e.,  the  density  must  satisfy  the  forward 
equation  of  Kolmogorov13 

2 

at  Pty^l*)  =  )  dfjy-  (a^CyipCy.tlx)) 

i»j=l  i  J 


~  2  r—  (b  .  (y)p(y ,  1 1  x)  ) 
j=l  yj  3 


p(y ,0  x )  =  6(y  -  x). 


WELL  POSED  PROBLEMS 

For  both  the  forward  and  backward  equations  we  have  only  imposed  initial 
conditions  which,  in  general,  are  not  sufficient  to  assure  a  unique  solution. 
We  must  introduce  either  boundary  conditions  or  some  other  condition  in  the 
absence  of  a  boundary  to  match  up  a  solution  of  the  PDEs  with  the  stochastic 
differential  equation.  This  report  treats  in  detail  only  examples  from  the 
no-boundary  case. 

The  best  introduction  to  the  problem  of  appropriate  boundary  conditions 
is  Feller's  paper  on  one-dimensional  diffusions.14  16  The  scope  of  the  prob¬ 
lems  is  discussed  here. 

Consider  the  problem  of  modeling  a  constrained  pendulum;  i.e.,  a  simple 
pendulum  with  a  tagline  limiting  the  pendulum's  arc.  On  the  free  portion  of 
the  arc  the  system  might  be  modeled  by 

x  +  ax  +  bx  =  f(t) 

x  >  x  , 
o 


12ibid . 

13ibid . 

14ibid. 

16Feller,  W. ,  Diffusion  Processes  in  One  Dimension,  Trans.  Amer.  Math.  Soc., 
77(1954),  pp.  1-31. 
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where  f(t)  represents  an  exogenous  disturbance.  In  vector  lorin  the  model 
becomes 


ji(t)=[-b  -I]  x(t>  ♦[?<,)] 

1  X  (t)  >  X 

1  o 

Modeling  the  disturbance  as  white  noise  the  stochastic  d i  f  (emit  ia  1  equation 
becomes 


dX(t)  =  BX(t)  dt  +  a  dW(t) 


where 


r? 

and  a  = 

\n 

°1 

i_-b 

-aj 

l_o 

£  J 

The  backward  equation  is 


3u 

3t 


aif 


+  X, 


3u 

3x 


-  (bXj  + 


1 


ax2) 


3u 


The  model  is  obviously  incomplete  since  we  have  not  specified  what 
happens  on  the  boundary  x^  =  Xq.  If  we  want  to  model  the  process  as  a 

diffusion,  then  the  boundary  must  be  modeled  as  an  absorbing,  reflecting, 
or  elastic  barrier,  or  the  process  at  the  boundary  must  be  modeled  as  an 
elementary  return  process.16  We  stop  here,  but  the  point  to  be  made  is 
that  the  theory  includes  a  very  rich  class  of  models. 

We  are  getting  very  deep  into  the  theory  of  dynamical  systems;  however, 
we  must  continue  a  bit  further  to  get  a  feeling  for  how  all  ol  the  pieces 
fit  together.  Recall  the  relation 


00 

u ( t , x )  =  f  f(y)p(y,t|x)  dy. 
-00 


Since  u  depends  on  f,  t,  and  x,  let  us  rewrite  the  relation  as 


00 


[ T t f ] ( x )  =  f  f(y)p(y,t|x)  dy 
-00 


Thus  for  each  function  f  we  can  think  of  T^f  as  a  function;  i . e .  ,  T  is  a 
function  from  some  set  of  functions  to  a  set  of  functions.  |,ct  us  call  this 


,6ibid . 
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set  G.  So  far  each  t,  T  is  a  function  from  G  into  G  and  {Tt,t  >  0}  is  called 
a  semigroup.17 

The  study  of  semigroups  has  a  large  literature  in  its  own  right.  The 
modern  theory  of  Markov  processes  is  largely  semigroup  theory.7  Of  interest 
to  us  is  that  the  study  of  the  forward  and  backward  questions  is  subsumed 
in  semigroups  and  the  problem  of  appropriate  boundary  conditions  or  other 
conditions  specifying  the  solution  for  the  PDEs  when  the  boundaries  are 
absent  is  translated  into  conditions  on  the  space  G.12 


ASYMPTOTIC  RESULTS 


The  qualitative  analysis  of  dynamical  systems  is  largely  the  description 
of  the  long  term  behavior  of  the  system.  For  deterministic  systems,  this 
involves  the  determination  of  the  stable  points  of  the  system.  The  behavior 
of  stochastic  systems  is  much  richer;  instead  of  stable  points,  we  look  for 
steady-state  distributions  or  densities.18  Since  we  usually  deal  with  systems 
whose  state  vector  takes  on  every  permissible  value  infinitely  often,  we  can¬ 
not  talk  of  stable  points  but  only  of  the  relative  amount  of  time  the  state 
vector  is  nearby  some  fixed  point. 

Obtaining  asymptotic  results  for  the  backward  equation  seems  to  be  easier 
than  for  the  forward  equation.  They  can  then  be  translated  into  results  for 
the  forward  equation.  Furthermore,  we  seem  to  have  a  better  idea  of  what  to 
look  for;  i.e.,  we  want  u(t,x)  to  approach  a  constant  function  as  t  ■*  ®. 
Therefore,  the  expected  value  of  f(X(t)),  as  t  ■*  »,  is  independent  of  the 
starting  value  of  X(t)  ,  a  reasonable  way  for  physical  systems  to  behave. 

We  only  have  partial  results  to  report  in  this  area,  so  treatment  is 
left  a  little  skimpy.19  However,  the  results  do  show  in  outline  how  semigroup 
theory  is  used. 


Consider  the  one-dimensional  problem 


3u 

at 


h  a(x) 


a2u 

ax5 


+  b(x) 


3u 

ax 


7ibid . 

12ibid. 

17Curtain,  R.  F.  and  Pritchard,  A.  J. ,  Infinite  Dimensional  Linear  Systems 
Theory,  Springer-Verlag,  1978. 

18Papanicolaou,  G.  C.  and  Kohler,  W. ,  Asymptotic  Analysis  of  Deterministic 
and  Stochastic  Equations  with  Rapidly  Varying  Components,  Comm.  Math.  Phys., 
45  (1975),  pp.  217-232. 

19Brannan,  J.  and  Reneke,  J.,  Unpublished  Research,  1981. 
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on  -»  <  x  <  Ob,  We  seek  conditions  on  a(x)  and  b(x)  which  ensure  that  u(t,x) 
has  a  constant  limit  as  t  ■* 

One  such  set  of  conditions  is: 

1.  a(x)  and  b(x)  are  continuous  on  (-»,<»)  and  a(x)  >  0 
for  all  x. 

2.  The  function  f(x)  defined  by 

x 

f(x)  =  exp(2  /  b(s)/a(s)  ds)/a(x) 

0 

is  absolutely  summable  on  (-»,  <») ;  i.e., 

00 

J  |f(x)  |dx 
-00 


exists  and  is  finite.  For  example,  if  dX(t)  =  -X3(t)  dt  +  £  dW(t), 
then  the  expected  value  of  X(t)  as  t  ■*  “>  is  independent  of  X(0). 

Let  k  and  m  denote  the  increasing  functions  defined  on  (-<»,  °°) ,  respec¬ 
tively,  by 


t  x 

k(t)  =  J  exp  (-2  J*  b(s)/a(s)  ds)  dx 
0  0 


and 


t  x 

m(t)  =  /  2  exp  (2  J  b(x)/a(s)  ds)/a(x)  dx. 

0  0 

Let  G  denote  the  complex  valued  functions  f  on  (-»,  »)  such  that  each  of 

Oo  00 

f  j f 1 2  dm  =  f  f f (x ) [ 2  m’(x)  dx 
-00  -00 


and 


Oo  00 

;  jdf |2/dk  =  ;  |f(x)j2/k’(x)  dx 


-Oo 


-00 


exists  and  is  finite.  Let  <•,•>  denote  the  inner  product  defined  on  G  by 

00  00 

<f,h>  =  f  fhdm  +  J  df  dh/dk. 

-00  -00 
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A  consequence  of  the  Hille-Yoshida  Theorem17  is  that  we  can  define  a 
semigroup  {Tt,  t  >  0}  on  G  by  [Ttf](x)  =  u(t,x)  provided 

3u  i  /  32u  .  ur  \^u 

at  =  *  a(3°  +  b(x)ax 

u(0,x)  =  f(x). 

(There  is  only  one  solution  u(t,x)  of  the  PDE  such  that  u(t,*)  is  in  G  for 
each  t  >  0.)  Furthermore,  <u(t,*),  u(t,*)>  <  <f,f>  for  each  t  >  0;  i.e., 
{Tfc,  t  >  0}  is  a  contraction  semigroup. 

We  can  conclude,  finally,  that  {T^f}  has  a  limit  in  G  as  t  ■*  ®  and  that 
the  limit  is  constant.  Let 


Then 


1 

p(t,x)  =  j^y  exp (2  /  b(x)/a(s)  ds)  u(t,x)  . 


~  p(t,x)  =  ^y  exP  (2  /  b(s)/a(s)  ds)  g|  u(t,x), 

(a(x)  p(t , x) )  =  ~y~  exp  (2  J  b(s)/a(s))  ds)  u(t,x) 


+  exp 


(2 


J  b(s)/a(s) 
0 


ds)  u(t,x) 


=  2  b(x)  p(t,x)  +  exp 


(2  J  b(s)/a(s)  ds)  —  u(t,x), 
0  3x 


and 

(a(x)  p(t,x))  =  2  ~  (b(x)p(t ,x)) 

+  2  exp  (2  J  b(s)/a(s)  ds)  —  u(t,x) 

x  92 

+  exp  (2  J"  b(s)/a(s)  ds)  u(t,x). 


17ibid. 
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^7  p(t,x)  -  \  (a(x)p(t,x))  +  ~  (b(x)p(t,x)) 


-(7)  exp  (2  f  b(s)/a(s)  ds)  ~  u(t,x)  -  y-  (b(x)p(t,x)) 


a(x)  eXp  J  b(s)/a(s)  ds)  u(t,x) 


*  a2  a 

-  *5  exp  (2  J  b(s)/a(s)  ds)  yyj  u(t,x)  +  --  (b(x)p(t,x)) 


^-~y  exp  (2  J  b(s)/ a(s)ds)  {yy  u(t,x)  -  a(x)  u(t,x) 


-  b(x)  g—  u(t,x)}  =  0. 


Furthe rmore , 


P(t,x)  -»•  exp  (2  J  b(s)/a(s)  ds) 


as  t  -*  00  for  some  constant  c. 


SECTION  IV 


SYSTEM  IDENTIFICATION 


DEVELOPMENT  OF  LINEAR  EQUATIONS 

For  the  method  of  moments  in  its  simplest  form,  one  tries  to  choose  the 
parameters  so  the  model  moments  approximate  the  empirical  moments.  For  the 
method  outlined  here,19  we  establish  some  relations,  linear  in  the  parameters, 
which  have  model  moments  as  coefficients.  We  then  substitute  the  empirical 
moments  in  place  of  the  model  moments  and  solve  in  a  least  squares  sense  for 
the  parameter  estimates.  In  the  case  of  boundaries,  the  relations  probably 
will  have  to  include  some  which  are  nonlinear. 


19ibid . 
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Interchanging  the  roles  of  x  and  y  yields 


NCSC  TM  339-82 


J  /  b  (x,y)  p(x,y)  dxdy  =  0. 
-oo  -oo 


For  i  and  j  non-negative  integers,  let 

00  00 

m. .  =  J  J  x1yJ  p(x,y)  dxdy. 
XJ  -00  -00 


In  the  linear  case,  the  equations  m  ^b^  +  “gjb^  =  ®  contain  no  information 
since  m1Q  =  n>01  =  0.  However,  we  can  repeat  the  above  process,  but  first 
multiplying  by  either  x  or  y;  i.e., 


x  y  a2r  =>2* 


0  =  e2  J  J  {u  |^§(u,v)  +  u  0(u,v)}  dudv 


3yz 


x  y  3(b,p)  3(b  p) 

-I  /  {u  — (u,v)  +  u 


-oo  -oo 


(u,v)}  dudv 


y  x  - 

=  h  e2  J  {x  g^(x,v)  ■  P Cx , v) }  dv  +  %  e2  J  u  ^(u,y)  du 


3y 


J  {x  b1(x,v)  p (x , v)  -  /  b1(u,v)  p(u,v)  du}  dv 


-  /  u  b2(u,y)  s (u , v)  du. 
-00 


Let  y  go  to  ».  Then 


0  =  *  e 2  /  {x  |^(x,y)  -  p(x,y)}  dy 
-00 


-  /  {x  bjCx.y)  p(x,y)  -  J  bj(u,y)  p(u,y)  du}  dy. 


Integrating  with  respect  to  x  from  to  »  we  have 
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0 


00  06 

h  £2  X  /  {x  |^(x,y)  -  p(x,y)}  dxdy 
—00  —00 


oo  oo  X 

-  X  X  {x  b1(x,y)  p(x,y)  -  /  b  (u,y)  p(u,y)  du}  dydx. 

-00  — 00  -oo 

In  the  linear  case  this  yields  m2Q  b^  +  b^2  =  -h  e2. 

On  the  other  hand,  if  we  let  x  go  to  oo  and  integrate  with  respect  to  y 
from  -oo  to  o»  we  obtain 

00  00  00  00 

o  =  X  X  y  b,(x,y)  p(x,y)  dxdy  +  f  f  x  b  (x,y)  p(x,y)  dxdy. 

-00  -00  -00  -00 

In  the  linear  case,  n>n  bn  +  mQ2  bJ2  +  m2Q  b2J  +  m11  b22  =  0. 

As  before,  interchanging  the  roles  of  x  and  y  we  also  obtain  the  equation 

00  00 

X  X  y  b  (x,y)  p(x,y)  dxdy  =  -%  e2  . 

-00  -OO 

We  can  either  use  higher  moments  to  obtain  enough  equations  or  some  other 
knowledge  of  the  model.  For  instance,  if  the  model  is  symmetric,  i.e., 
b^2  =  b21,  we  obtain  the  following  system: 


— 1 

5 

N) 

O 

■n 

0 

0 

*-* 

_ 1 

-h  e2 

mn 

”02 

m20 

mll 

b12 

0 

0 

0 

"11 

m02 

b21 

e2 

0 

1 

-1 

0 

I 

CM 

CM 

jVi 

0 

Example  1 

Suppose  e  =  1  and  the  moments  are  given  as  m^  =  =  0,  m2Q  = 

and  “jj  =  ®Q2  =  V*  The  system  above  becomes 


- 1 

■jr 

«■ r 

o 

o 

J 

~-h  ~ 

\  \  \  k 

b12 

0 

0  0  k  k, 

b21 

-h 

_0  1  "2  °_ 

-b22  - 

0 
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which  is  seen  to  be  satisfied  by  b  =  -2,  b  =  b  =  2,  and  b„  =  -A.  The 
model  is  11  1Z  11 

dX(t)  =  [_2x  -  Ay]dt  +  dW(t)‘ 


EXAMPLES  OF  FLOWS  WITH  POTENTIALS 


We  cannot  at  present  solve  the  equation 


\  e2 


a?p  a(bip) 

By2  "  3x 


3(b2p) 

8y 


=  0 


for  a  non-negative  summable  function  p.  In  the  special  case  that 
[bjCx.y)  b2(x,y)J  has  a  potential  function  $,  i.e.,  8$/8x  =  b1  and 

3<fr/8y  =  b2>  then  p(x,y)  =  c  e2^x»y)f  where 

OS  00 

c  =  1  / CJ  /  exp  (2<Kx,y))  dxdy) 


is  the  unique  steady-state  density.  For  this  case,  we  can  construct  numerical 
examples  avoiding  simulations. 

Example  2 

Consider  the  nonlinear  model 


dX(t)  =  3X  "  y]  dt  +  £  dW(t) . 


The  potential  function  is 

<Kx,y)  =  -x4  +  3/2  x2  -  xy  -  \  y 2  . 

Clearly, 


lim  xV^*'^  -  0  and 

X->±oo 


.  .  i  2<>(x,y) 
lim  yeT 

y-»±0o 


=  0 


uniformly  on  bounded  sets  for  every  i.  Hence  we  can  use  the  density 

p(x,y)  =  e2<^x,y^  to  generate  some  moments  numerically  and  test  the  method 
outlined  above.  For  this  model,  m1Q  =  mQ1  =  m3Q  =  m21  =  mJ2  =  mQ3  =  0. 

So  in  order  to  have  enough  linear  relations  to  determine  the  "unknown" 
parameters  in  the  model 
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dX(t)  = 


+  2a^x  +  a^y 
a3x  +  2a4y 


dt  +  e  dW(t) , 


we  must  use  at  least  fourth  order  moments  (i  +  j  =4). 

Besides  the  equations  generated  before  we  add 
os  os 

/  /  y3b2(x>y)  p(x.y)  d*dy  =  -3/2  e2  mQ2  . 

—  00  —00 

These  relations  would  give  four  equations .  We  can  add  two  more  by  assum¬ 
ing  that  there  are  modes  at  (1,  -1)  and  (-1,  1);  i.e.,  bjO,  -1)  =  0  and 

b^C-l,  1)  =  0.  Thus  we  have  (assuming  e  is  known)  the  following  system: 


~4 

2 

-1 

0 

ai~ 

"o 

0 

0 

1 

-2 

a2 

= 

0 

0 

0 

“'ll 

2m02 

a3 

c2 

4m40 

2m20 

mil 

0 

a4 

-h  e2 

°"20  +  D,02) 

—  — 

4m31 

2mil 

2 

2mll 

0 

0 

0 

mi3 

2m04_ 

-3/2  e2  m02  _ 

Since  the  coefficient  matrix  is  known  only  approximately,  we  can  increase 
the  accuracy  of  our  estimates  by  adding  relations.  As  indicated  above,  the 
additional  equations  need  not  involve  moments  but  may  arise  from  some  other 
knowledge  we  have  of  the  model. 


SECTION  V 

QUALITATIVE  ANALYSIS 


MULTIPLE  MODES 

The  qualitative  analysis  of  nonlinear  stochastic  systems  is  complicated 
by  the  possibility  of  steady-state  distributions  which  are  not  Gaussian.  Con¬ 
sider  the  one-dimensional  model 


dX(t)  =  -X(t)(X2(t)  -  1)  dt  +  dWt. 
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The  forward  equation  is 


l)p) 


and  steady-state  is  given  by 

Ijp"  +  [x(x2  - 

Thus 

x 

p(x)  =  ce-2  J  s3  - 
o 

Note  tha£  p'(x)  =  (-2x3  +  2x)  p(x)  =  0 
critical  points  can  be  classified  as: 
a  local  minimum.  Thus  p(x)  is  the  density  of  a  bimodal  distribution. 

The  system  must  linger  proportionately  longer  in  neighborhoods  of  states 
-1  and  1  than  in  neighborhoods  of  0.  Thus  the  system  might  exhibit  a  kind  of 
periodic  motion  visiting  alternately  neighborhoods  of  states  -1  and  1  with 
some  expected  time.  If  the  system  were  deterministic  (eliminating  the  noise 
term),  then  the  trajectory  upon  arriving  in  certain  neighborhoods  of  either 
-1  or  1  would  not  exit.  The  long  term  behavior  of  Markov  processes  requires 
a  descriptive  apparatus  substantially  different  from  the  stability  analysis 
of  deterministic  systems. 


Dp]'  =  o. 


.  -x4/2  +  x2 

s  ds  =  me 


has  solutions  x  =  -1,  0,  1.  These 
-1  and  1  are  local  maxima  and  0  is 


FIRST  PASSAGE  TIMES 

Consider  the  one-dimensional  model  dX(t)  =  b(X(t)  dt  +  e  dW(t)).  Suppose 
x  <  0.  We  want  to  compute  the  probability  that  X(t)  >  B  given  X(0)  =  x.20 
In  order  to  do  this,  we  treat  f)  as  an  absorbing  boundary  and  so  we  must  pause 
to  discuss  the  corresponding  boundary  condition.15  21 

Let  F(y,t|x)  =  Prob(X(t)  <  y|X(0)  =  x)  denote  the  conditional  probability 
distribution  function  and  J(y,t|x)  the  rate  at  which  probability  is  flowing 
in  the  positive  y-direction  at  y  at  time  t.  Then  J  =  -9F/9t  is  called  the 
probability  flux  and  9p/9t  =  32F/9t3y  =  -9J/9y;  i.e., 

J(y»t|x)  =  b(y)  p(y,tjx)  -  h  a  gy  p(y,t  x) . 


15ibid. 

20Matkowsky,  B.  J.  and  Schuss,  Y. ,  The  Exit  Problem  for  Randomly  Perturbed 
Dynamical  Systems,,  SIAM  J.  Appl.  Math.,  33(1975),  pp.  365-382. 

21Friedman,  A.,  Stochastic  Differential  Equations  and  Applications,  Academic 
Press,  1975. 
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The  concept  of  probability  flux  is  useful  in  formulating  the  appropriate 
boundary  condition. 

With  an  absorbing  barrier  at  {3  the  Chapman-Kolmogorov  equation  becomes 

P 

p(z i t  +  s | x)  -  J  p(z,t|y)  p(y,s[x)  dy. 

-00 

As  before,  if  u(t,x)  is  any  solution  of  the  backward  equation  then 

P  P 

/  u(0,z)  p(z,t  +  T | x)  dx  =  /  u(t,y)  p(y,t|x)  dy. 

-00  -00 

One  notes  that  differentiating  the  left-hand  side  wrt  t  or  T  is  equivalent. 
Hence,  by  virtue  of  the  equality,  differentiating  the  right-hand  side  by  t 
or  t  is  also  equivalent.  Thus 


P  g  P  a 

S  u(t,y)  3^  p(y,t|x)  dy  =  J  ^  u(t,y)  p(y,t|x)  dy 

-00 


(Substituting  the  expression  for  3u/3t  from  the  backwards  equation  into  the 
right-hand  expression,  one  has) 


P  a2  a 

J  a  u(t,y)  +  b(y)  ^  u(t,y)]  p(y,t|x)  dy 


(*) 


(Integrating  by  parts  leads  to) 


h  a(g^  u(t,y))  p(y , t | x) 


y  =  i  +  b(y)u(t,y)  p(y,t|x)  J*  =  _£ 


P 

-  J  {*1  a  gy  p(y,t|x)  u(t,y)  +  gy  (b(y)  p(y,t|x))  u(t,y)}  dy 

(The  probability  density  is  zero  at  -•»  and  (J;  integrating  the  first  term  in 
the  integrand  by  parts  one  has) 


-h  a  g|  p(y,T i x)  u(t,y) 


P 

-00 


+  /  a  g^2  p(y,t|x)  -  gy  (b(y)  p(y,t|x))}  u(t,y)  dy 


(Substituting  from  the  forward  equation  the  expression  for  3p/3t  into 
equation  (*)  and  comparing  with  the  expression  directly  above,  one  has) 
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m 


or  J(p,t  x)  u(t,p)  =  0.  Since  in  general  J(p,t|x)  j*  0,  then  we  must  require 
that  u(t,p)  =  0,  our  boundary  condition  for  the  backward  equation. 

Thus  the  probability  that  X(t)  <  P  given  X(0)  =  x  is  found  by  solving 

ai  =  *  a  WF  +  b(y)  3y 

s  j  1  for  y  <  p 
U(0’y)  =  >0  for  y  >  p 

u(t,p)  =  0 

Further,  1  -  u(t,x)  is  the  proportion  of  trajectories  that  reach  P  in  [0,t] 
and 

-  g|  u(t.x)  =  J(P,t jx). 

Let  w(t,x)  =  -3/3t  u(t,x).  Then  differentiating  the  backward  equation 
yields 

3w  ,  32w  .  ,  .  3w 

ai  =  11  *  +  b(x)  3i  • 

w(t,P)  =  6(t), 
w(t  ,-<*>)  =  0, 

and  w(0,x)  =  6(p  -  x) 

0O 

and  J  w(t,x)  dt  is  the  proportion  of  trajectories  which  leave  (-«®,p)  eventually. 


T  =  inf  {t:  X(t)  =  p,  X(0)  =  x} 
the  exit  of  first  passage  time  and 


z(x)  =  E(T)  =  /  tw(t,x)  dt. 


If  Prob  (T  <  •)  =  l  or,  equivalently 


J  w(t,x)  dt  =  1, 
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then 


/  t  w(t,x)  dt  =  »s  a  /  tw(t,x)  dt  +  b(x)  S  tw(t,x)  dt 
o  0  0 


=  t  w(t,x) 


00 


o 


00 

f  w(t,x)  dt 
o 


or 


h  a  z"  +  b(x)  z'  =  -1,  z(p)  =  0,  and  z'  (-»)  =  0. 


Examples 

We  want  to  compare  the  first  passage  times  from  0  to  2  for  the  following 
models : 

1.  dX(t)  =  -2  X(t)  dt  +  £  dW(t) 

2.  dX(t)  =  -(X3(t)  -  3  X2(t)  +  2X(t))  dt  +  £  dW(t) 

3.  dX(t)  =  -(2X3(t)  -  5  X2(t)  +  2X(t) )  dt  +  £  dW(t) 

4.  dX(t)  =  -(4X3(t)  -  9  X2(t)  +  2X(t))  dt  +  £  dW(t). 

Mote  that  b(x)  has  zeros  at  0  and  2  for  each  of  2  through  4  and  a  zero  between 
0  and  2  which  is  nearer  to  0  for  4  than  for  3  and  nearer  for  3  than  for  2. 

The  expected  first  passage  times  for  the  four  models  are  approximately  500, 

4 ,  2 ,  and  1 . 

In  all  four  models,  the  tendency  for  a  trajectory  in  a  small  enough 
neighborhood  of  0  is  to  return  to  0.  For  the  last  three  models,  a  trajectory 
to  the  right  of  the  middle  zero  of  b(x)  has  a  tendency  to  go  to  2.  The  closer 
that  zero  is  to  0  the  easier  for  the  random  disturbance  to  push  the  systm  into 
the  region  of  attraction  of  2. 


SECTION  VI 
CONTROL  PROBLEMS 


FEEDBACK  CONTROL 

The  first  objective,  using  an  analogy  with  linear  systems,  is  to  control 
the  given  system  so  the  steady-state  distribution  is  unimodal  with  mode  at  0. 
A  second  objective  might  be  to  fix  moments  higher  than  the  first. 
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Consider  a  one-dimensional  model  dX(t)  =  b(X(t))  dt  +  e  dW(t).  If  the 
model  is  linear,  i.e.,  b(x)  =  bx,  then  the  question  of  stabilizing  the  system 
is  trivial.  Looking  at  the  deterministic  system  dX(t)  =  (b  •  x(t)  +  u(t))  dt 
we  can  choose  u(t)  =  c  •  x(t)  where  b  +  c  <  0.  In  the  nonlinear  case  if  we 
require  u(t)  to  be  of  the  form  u(t)  =  c  •  x(t),  there  might  be  no  such  u(t) 
which  stabilizes  the  system.  However,  even  if  we  require  that 

dX(t)  =  b(X(t))  dt  +  e  dW(t) 

has  a  steady-state  density  we  can  wonder  if  there  is  a  number  co  so  that  the 
steady-state  density  of 

dX(t)  =  (b(X(t))  +  c  X(t)  dt  +  e  dW(t) 

is  unimodal  with  mode  at  0. 

The  following  examples  illustrate  the  answer  to  this  question. 

Example  1.  The  scaler  nonlinear  system 

dX(t)  =  [-X(t)  (X2(t)  -  1)  +  c  X(t)]  dt  +  dW(t) 
has  steady-state  density 

p(x)  =  m  exp (-x- 4/2  +  (1  +  c)  x2). 


Further, 

p'(x)  =  p(x)  ♦  (-2x3  +  2  (1  +  c)  x) 

has  exactly  one  zero  if  c  <  - 1 .  Clearly,  this  generalizes  if  b(x)  can  be 
differentiated. 

Suppose  the  two-dimensional  model 


dX(t) 


■b^XCt))' 
.b  (X(t))_ 


dt  +  e  dW(t) 


has  a  potential  and  b^(0,0)  =  b2(0,0)  =  0.  Sufficient  conditions  on  a  2x2 
matrix  C  so  * 


dX(t)  = 


bj(X(t))  +  'cnX1(t)  +  c12X2  (t) 
[b2(X(t))  +  c21Xj(t)  +  c22X2(t)  J 


dt  +  e  dW(t) 


has  a  mode  at  (0,0)  are 

1 .  c  =  c 
*  12  21 
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3b? 

2. 

C11  < 

3x 

(0,0) 

3b2 

3. 

V 

M 

C*J 

u 

By 

(0,0) 

3b 

3b  _  3b, 

4. 

C°.°)  +  cn]  (°,0)  +  C22]  "  ^~3y  <°*°>  +  C12]2  >  ° 

Example 

_2.  Consider 

the  system 

'-4X?  (t)  +  3X  (t)  -  X  (t)' 
dX(t)  =  dt  +  e  dW(t) 

-X  (t)  -  X  (t) 


whose  steady-state  density  has  local  maxima  at  (1,  -1)  and  (-1,  1).  The 
indicated  partials  are  Sbj/Bx  =  -12  x2  +  3,  Bbj/By  =  -1,  3b  /3x  =  -1,  and 
Sb^/By  =  -1.  Hence  sufficient  conditions  are  c^  <  -3,  c22<  1 »  C12  =  C21* 

and  (3  +  c^H-l  +  c22)  -  (-1  +  c12)2  >  0.  If  we  choose  cJ2  =  c2J  =  0, 

=  -4,  and  c22  =  0  then  b^  =  -4x3  -  x  and  t>2  =  -y.  For  these  choices, 
the  controlled  system 

f-4X3  (t)  -  X.  (t)l 

dX(t)  =  L-X2  (t)  J  dt  +  e  dW(t) 

has  unimodal  steady-state  distribution  with  mode  at  (0,0),  The  question  of 
whether  this  result  can  always  be  obtained  is  open. 

No  optimal  control  problem  has  been  formulated. 22  23  The  formulation 
of  a  significant  optimization  problem  must  be  preceded  by  the  formulation 
of  reasonable  objectives.  This  in  turn  requires  more  insight  into  the 
qualitative  analysis  for  nonlinear  stochastic  systems. 


22Fleming,  V.  H.  and  Rishel,  R.  V.,  Deterministic  and  Stochastic  Control, 
Springer-Verlag,  1975. 

23Kushner,  H.  J. ,  Stochastic  Stability  and  Control,  Academic  Press,  1967. 
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APPENDIX  A 

THE  LINEAR  RELATIONS  FOR  THE  PARAMETERS 
IN  TERMS  OF  THE  MOMENTS 


The  starting  point  for  obtaining  linear  relations  for  the  parameters  of 
the  n-dimensional  system  is  the  steady-state  forward  equation. 


0  =  h  I  a.  . 


_afp_ 


n  3(b.p) 


ij  9x  dx  8x . 

1»J_1  1  J  J=1  j 


Multiplying  by  (x  )m  and  integrating  x.  from  -oo  to  oo  for  i  =  1.2  n  we 
obtain  K  1  ’  ' ’  ’ 


0  =  %  I  a  J1  ...  (uj“ 
i,j=l  J  -«  -o» 


k>  ^TaT  (u)du 

1  3 


®  x,  _  a(b.p) 

“  1  /  •  •  •  /  (uR)  — —  (u)du 

j  =  l  -00  -00  K  dXj 

Let  x4  -*■  <*(i  f  k)  and  integrate  from  -•  to  “  with  respect  to  x.  .  Then 


°  \£  ik  <“")  ''*«■  Sif  '  (,1 . “k . 


)du^dx 


2k  {  L  (xk)m  a?(x)dx  "  "  L  (uk> " _1  a?  <V  •••»  V  ’  xn)dukdx> 


x 

Rn  -oo 

X 
/ 

R“  -oo 


'  Jn(V"  bk(x)  P<x>dx  +  “  4  [bkpJCxi . , 

R  R  -00  Ik  i  K 


8  )dl! 


dx 


-  (1  +  m)  /n  (*k)m  bk(x)p(x)dx 
R 

m(m  *f  1 )  „  B*i  m 

'  2  *kk  4  (xk}  P(x)dx  -  O  ♦  «)  Jn(*k)  bR(x)p(x)dx. 

^  R 
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Or 


Su  <V"  bk(x)p(x)dx  =  -  f  akk  /n  (xk)“'1  P(x)dx. 

R  ** 

On  the  other  hand,  for  i  t  k  let  x.  -»  »  (j  #  i)  and  integrate  from  -» 
to  00  with  respect  to  x^. 


-  L  £  <xk>*  sfe  <xi-  -  v  •  Vduidx 

K  -® 


X. 


i  k 
xm  32 


* '  akk  £  fl  (\r  s?  <-i . “i . x«)d"idx 

R  -00  1 


X  . 


*  *  *ii  £  £  ‘V  S*  <V  •  •  •  “l . Xn)d“idx 

Xi  m  3(biP) 

■L  I  <V  T5T- (V  •  “i . x»)d“idx 

R  -w  i 


x.  _  8(b.p) 


L,  ^  <xk>'  “ST  <X1 . “i . V  d“idX 

R  -»  k 

=  -•  *ik  £  <xk,"‘I  p(x)dx  ‘  1  %k  £  X  <xi'  ••••  ”i' 

*  ■>  “u  u  <V‘  sE<x>dx  -  (xk>”  bt(x>p(x,dx 


R 
x . 


+m  J  (x^)  Jb^p]  (Xj,  •••*  ut,  ...»  x^Jdu^dx. 

If  m  =  l,  then 

0  =  a..  +  x  b. (x)p(x)dx  +  J  x  b. (x)p(x)dx. 
IK  pH  K  1  p11  1  R 

If  m  >  1,  then 


x  )du.dx 
n  l 


A- 2 
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maik  Rn  ^k5"'1  P(x)dx  +  m(V  ^  3kk  [n  £  (xk)m"2  p(V 


.m- 1 


X  (x.)  b . (x)p(x)dx  -  m  f  (x  )  x  b  (x)p(x)dx. 
Rn  X  1  Rn 


.m-1 


/  (x.)  b . (x)p(x)dx  +  m  J  (x  )  x  b  (x)p(x)dx 

tn  K  x  k 

K  A 


"m  aik  L  (xk)m_1  p(x)dx 

R 


-  1)  f  ,  ..in-2 

- - -  a.  ,  X  (x,  ) 

2  kk  Rn  k 


x^xjdx 
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